Papers by Wietse de Vries
What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | In addition, information on part-of-speech tagging is spread over different parts of the network and the pipeline might not be as neat as it seems. |
| Approach: | They propose to probe Dutch BERT-based model and multilingual BERT model for Dutch NLP tasks to see if this holds true for other languages. |
| Outcome: | The proposed model is based on a Dutch model and a multilingual model for Dutch NLP tasks. |
Adapting Monolingual Models: Data can be Scarce when Language Similarity is High (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained language models are the dominant approach for solving many tasks in natural language processing. |
| Approach: | They propose to retrain the lexical layers of four BERT-based models using data from two low-resource target languages while the Transformer layers are independently finetuned on a POS-tagging task in the model's source language. |
| Outcome: | The proposed method achieves high performance for both target and target languages with high similarity. |
Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies on cross-lingual generalisability of large pre-trained models use English training data and test data in unseen languages. |
| Approach: | They propose to use multilingual pre-trained models to model cross-lingual transfer in a selection of target languages. |
| Outcome: | The proposed model can be used to improve cross-lingual transfer performance in low-resource languages with no labeled training data. |
DUMB: A Benchmark for Smart Evaluation of Dutch Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Current Dutch monolingual models under perform and suggest training larger models with other architectures and pre-training objectives. |
| Approach: | They propose a Dutch Model Benchmark that compares performance of language models to a strong baseline that can be referred to in the future even when assessing different sets of language model. |
| Outcome: | The proposed benchmark compares the performance of 14 pre-trained language models to a strong baseline . the results suggest training larger models with other architectures and pre-training objectives . |
As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing pre-trained language models are limited in their ability to train for English, which is a problem for many other languages. |
| Approach: | They propose to adapt existing generative language models to new languages by retraining lexical embeddings without tuning the Transformer layers. |
| Outcome: | The proposed method achieves lexical embeddings for Italian and Dutch that are aligned with the original English lexicals. |